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ABSTRACT 

This report presents the proceedings of a conference 
for researchers and trainers of researchers, convened to discuss the 
issues involved in improving knowledge by finding better ways to 
permit cross-research analyses or longitudinal research analyses. 
Included are: (1) a description of the history, goals, and activities 
of *the Interagency Panel on Early Childhood Research and Development 
and the Interagency Panel for Research and Development on 
Adolescence; (2) an address def inilig the need for comparability and 
generalizability in research and the problems involved; (3) summaries 
of the discussians of four separate conference workgroups; (4) 
post-conference reactions of some of the participants; (5) a brief 
description of the panels' conference followup activities; and (6) a 
short paper which summarizes the major recurring recommendation? of ^ 
workgroups from this conference and a preceding one. The paper 
suggests the organization of Research Progress Monitoring Grou^, a 
tripartite structure of interrelated groups designed to translate 
policy questions *into research questions and to attain answers to 
these questions as quickly as possible, (BD) 



J 



««««««««««« 4^ ««««««««« «4t«««««««««^ 

♦ Documents acquired by ERIC include many informal unpublished ♦ 

♦ materials not available from other sources, ERIC makes every effort ♦ 

♦ to obtain the best copy available. Nevertheless, items of marginal ♦ 
♦reproducibility are often encountered and this affects the quality ♦ 

♦ of the microfiche and hardcopy reproductions iSRIC makes available ♦ 

♦ via the ERIC Document Reproduction Service (EDRS^ , EDHS is not ♦ ' 

♦ responsible for the quality of the original document. Reproductions ♦ 

♦ supplied by EDRS are the best that can be madfe from the otiginjal, ♦ 

4^«««««4i«««V«««4i««««^«««« ««««««««♦«««««««♦♦♦♦♦♦♦♦♦ ♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦♦'^ 



ERIC 



Social Research Group 



^ The George Washington University U°dI?a^'on 11 weI^^^e'" - 

Washington, D.C. • * '*''^'°6ducTt^on^^°^ 

^-^^ , THt PE WiON CJk Ofcjfj^kNl/ATlON ORIGIN 

I ^ ATiNf, iT MOlNl', tjf VIE W fjR OPINIONS 

• ' 'JATtO DO N(jT Nl C E 'jSARIUV RtPttE 

, ' 'J N T Of f If tAL NATiONAl INSTl TUTF OF 

' * I Ou( A T lO^l PC/.iTiON f»R POL if Y 

Q ^ 



PROCEEDINGS OF THE SECOND CONFERENCE 
ON COMPARABILITY IN RESEARCH 

May 5 and 6, 1975 

« 

Sponsored by 

The Interagency Panel on Early Childhood 
Research and Development 

The Interagency Panel for Research 
and Development oil Adolescence 



00 

CO 
00 



Edited by: 

Maure Hurt, Jr., Ph.D. 
Thomas W. Hertz, Ph.D. 



The preparation of this publication was performed 
pursuant to contract -HEW 100-75-0010 with the 
Department of Health, Education, and Welfare 
Summer, 1975 



ERiC / 



Ackn^owledgments. 

The Office of Child Development, on behalf of the Interagency 
Panel on Early Childhood Research and Development and the Inter- 

V 

agency Panel for Research and Development on Adolescence, contracts 

with the Social Research Group of The George Washington University 

for svpport activities necessary to carry out the goals of the 

Panels. The Second Conference on Comparability in Research was a 

part of these support activities. 

As director of the project at the Social Research Group, I 

would like to take this opportunity to express my thanks to all those 

who contributed to the planning and organizing of the conference. 

^ / 

Among these are Panel nnembers who shared their time, skills' and experi- 

ence in the planning phases, as well as at the conference itself, es- 

V 

pjecially the Chairperson of the Panels, Edith Grotbierg; and also Social 
Research Group staff members' who gave, exceedingly careful attention to 
the sm^iad of details befor;Ey during, and after the conference, most 
notably Helen Somerville and Judy Miller, who were responsible for 
organizing the schedules, activities, facilities and accommodations, 
and for developing the materials for the Proceedings. 




Maure Hurt, Jr., ^ Ph.D. 
Social Research Group 



i 



TABLE OF CONTENTS 



Page 



WELCOME AND OPENl^NG REMARKS 1 

Edith H. Grotbefg, Ph.D. 

tHE NEED FOR COMPARABILITY IN* RESEARCH. . . .\ . . . 7 
Richard Q. Bell, Ph.D. 

SUMMARIES OF THE WORKGROUP DISCUSSIONS: 

WORKGROUP I ^. 15 

David Pearl, Ph.D., Moderator 

WORKGROUP II 22 

Richard Q. Bell, Ph.D., Moderator 

WORKGROUP III -31 

Carol McHale, Ph.D., Moderator 

^ WOJUCGROUP IV 39 

" Joseph M. Bobbitt, Ph.D., Moderator 

REFLECTIONS ON THE CONFERENCE v ... 47 

FOLLOW-ON ACTIVITIES , 51 

Edith H. Grotberg, Ph.D. 

WORKING pXpER: SYNTHESIS OF TOPICS FROM FIRST AND 
SECOND CONFERENCES ON COMPAJIABILITY IN RESEARCH • . . 53 
Richard Q. Bell, Pl^.D. 

ROSTER OF NAMES AND ADDRESSES:- 

UNIVERSITY REPRESENTATIVES AND OTHER GUESTS 
ATTENDING THE CONFERENCE 63 

MEMBERS OF THE INTERAGENCY. PANELS 67 



ERIC 



WELCOME AND OPENING REMARKS 

Edith H. Grotberg, Ph.D., Chairpersoa 
Interagency Panel on Early Childhoocf Research and Development 



We are d^ighted you could Attend this Second Conference on Comparability 
of Research. The first one was held in November and editors of the leading 
behavioral science and medical science journals attended. They discussed ' 
some of the same issues you will be discussing today and tomorrow, but they 
examined the issues from the point of view of editors, you will do it 

from the perspective of researchers and trainers of researchers. We are 
holding the Conferences because we need hfelp in Improving the cumula tiv^ess 
of knowledge through finding 'better ways to permit cross-research analyses 
or longitudinal research analyses. We feel you can help us here ^t the 
Federal Agency and Department levels and we trust we can help you as you 
grapple with problems of cumulative knowledge. But let me give you a quick 
historical sketch of who we are and what we wish to accomplish to improve 
research data for cumulative knowledge. We must begin with the establishment 
of the two « Interagency Panels. ^ 

The Interagency Panel on Early Childhood Research and D^elopment was 
established in 1970 as a resylt of the Secretary of the Department of Health, 
Education, and Welfare requesting one of the Agencies witl^fin the Dejp^rtment, 
the Office of Child Development, to take the lead in achieving greater coor- > 
dinatlon amon^ Agencies of research effort concerning childten. What 
prompted this request was the increased evidence that Agencies frequently 
duplicated research, overlapped research or ignored important gaps in research. 

The Director of the Office of Child Development called together representatives 

t, 

of various Agencies within. the Department who supported research on children.^ 



After a series of meetings and a good deal of discussion on structure, the 
Panel format v^s actfepted. This structure o^^ured the legislative and mission 
autonomy of each member Agency while providing a vehicle through which to 
promote the coordination of research planning and utilization. The Panel was 
not to a superordinate structure, but rather a point at which Agencies could 
meet for agreed upon actjfcyities. The Office of Child Development established 
an Information Secretariat, the head of which became Chairperson for the Panel. 
By 1972, the Secretary of the Department of Health, Educaticxp, and Welfare, upon 
the request of the Panel, invited other Depar^^ments to send representatives 
as new members to the Panel. The Department of Labor, the United States Depart- 
ment of Agriculture, and the Department of Housing and Urban Development responded 
to the invitation, and ^etame members. 

In 1972, also, a second Panel was established to address the problems of 
research "^concerning adolescence. Research staff from several Agencies who 
knew about the work of the Interagency Panel on Early Childhood Research and 
Development felt the need for similar kinds of work to solve similar kinds of 

problems relating to research on adolescence. The Director of the Office of ^ 

/ 

Child Development asked the Secretary of DHEW to invite Agencies and Departments 
to designate representatives to an Interagency Panel for Research and Develop- 
ment on Adolescence. The same Agencies and Departments responded but now 
designating a representative who specialized in research planning and development 
on adolescence. The first meeting w^s convened by# the Chairperson of the 

0 

Early Childhood Panel and the new Panel decided to adopt the organizational 
structure of the Early Childhood Panel as well as sharing the staff, the 
resources, and the Chairperson. 

The two Panels have developed an Information System which contains information 
on each research project fuiffiled by member Agencies. This information includes 



all on-going research as well as new starts and is updated annually . The 
information provides the basis for an Annual Report. Each year a report is 
published containing descriptions and analyses of current research funding 
activities of the Panels' member Agencies. The report includes charts and 
narrative, comparing what the Agencies are supporting in research. Other 
sections of the Annual Report address the problems of identif yiillg needk^ and 
gaps in research as well as determing where overlaps and duplications occur. 

The Panels frequently hold special interest meetings relating to an area 
of research in which many of the member Agencies have interest or are able by 
legislation to address. Outst^inding researchers present papers at these 
meetings and Panel members interact with the invited researchers- as well as - 
ajnong themselves. From these meetings the member Agencies draw ideas for 
interagency coordination of research planning and support. Follow-up meetings 
invite only those Agencies interested in coordination in the specif ic , research 
area. Some of the special interest meetings have addressed the following areas 
of research: (1) home-based programs; (2) longitudional/intervention research.; 
(3) work experiences; (4) the family; and (5) marker variables and marker 
measures. . ' 

In addition to coordination of research planning and support through the 
^device of special interest meetii^gs", the Panel members share the current and 
forward plans of their Agencies.' These plans are Included in the^'Annual Reports 
and are analyzed in tenna of the foci of the various Agencies/ From these 
current and forward plans the member Agencies are able to determine where they 
may wish to coordinate their research on an interagency basis. 

Not only da Agencies wish j^o use their resources well in terms of sup- 
porting research where needs are most critical 'and where unnecessary duplication 
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and overlapping are avoided, bat they also want to be -certain the research 
findings are usable. Research findings may be usable for application to problems, 
for ^ogram development, for services, but a critical use of research findings 
is/to compare -them with findings from other research. Cross^research and 
cumulative research analyses are meaningful only if there are some bases for 
comparability; i.e., the various research studies ar^ using comparable sub- 
jects, comparable definitions of variables, comparable measurement instruments. 
,As any researcher knows, it is very easy and often more productive for the 
researcher's own purposes to describe and define and measurfe wi^h new terms 
and new instruments created by the researcher. Probably the concern for 
imposing some ],i™its on researchers' creativity accounts for the fact that 
few Agencies addregs the problem of comparability. The United States De- 
partment of Agriculture is the major Agency among Panel members which supports ' 
cooperative research, a kind of research requiring all researchers who 

participate to agree on common definitions and measures. But for individual % 

\ 

research no Agencies, including USDA, requiyi^such agreements. 

To address this problem the Panels established a special interest group 
on marker variables and marker measures. T^e problem is complex, as the 

Panels soon learned. Many meetings of the special interest group have surfaced 

, I 
the many aspects of the problem, and several meetings have been held with 

foundations and professional organizations, such as the American Psychological 

Association, the American Educational Research A^sociatiqn, to involve them 

in discussions of the problem. A sub-committee of the Panels drafted a state- 

ment which set some guidelirffes for Agencies as well as foundations and 

professional associations which may serve to promote valid cros^-research and 

cumulative research analyses-. Thfese guidelines include requests for clearly 

stated definitions of population characteristics as well as an 
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inclusion of certain population characteristics, such as age, sex, ethnic 

*• 

identity, residence, etc. Some research does not need all of these population 
characteristics, but their inclusion permZ^s other researchers to conduct com- 
parative studies or analyses. 

The meeting today and 'tomori'ow is to address this problem of research 
comparability. We want a frank, open discussion and suggest that we break 
up this afternoon into smaller work groups so that everyone may speak. Each 
group is chaired by a member of one of the Panels and each group has a recorder. 
The recorders will work tonight and have summaries for us tomorrow morning 
so that each group may distribute and discuss with the entire group the 
deliberations. This is an important meetthg for us and we hope it will be 
seen that way by you. ^ 

Richard Bell, whom most of you know, and who has just left NIMH for the 
University of Virginia, has been very crucial in helping the Panels work on the 
problem of cumulative knowledge, marker measures and marker variables. He 
will present to you a statement on the problem from a researcher's point of 
view. Dick. * 
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THE NEED FOR COMPARABILITY IN RESEARCH 

Richard Q. BeU, Ph.D. 
The University of Virginia 



My interest in this area of comparability and g^eralizability came 
from a variety of experiences including those of, friends within and outside 
the Government. My personal experience came when I .found myself in the 
position of defending behavioral science research before administrators at 
the National Institute of Health. Some cf my friends had to go before Con- 
gressional committees to try to get money for researcH. Others in state, 
universities or agencies had to go before legislatures to get funds for 
research. The experience common to all of us was frustration at our inability 
to point to clear-cut, well-agreed upon findings in areas that had received 
support for years. 

One study frdm my experience further illustrates the problem. After a 
graduate student poured^hrough the literature for him, the investigator 
summed up results in that area for a 10 year period, covering a large numl)er 
of studies. Unfortunately' what the review came up with has come to be known 
as "Berelspn's findings." Berelson has caustically pointed out that research 
findings in the behavioral social sciences come down to this, typically: (1) 
the problem is more complicated than originally thought; (2) a larger sample 
is needed; and (3) more research is needed. V 

Obviously, these conclusions are not uniformly true across all areas of 
inquiry. I There h^ve ,been some ar^as of social science research in which giant 
strides have been^^iade, in which changes have come very rapidly. In other 
areas we have moved glacially. Paradoxically it seems as though these are 
ofteji the areas J^which the public need is greatest. There are many reasons 
why this happens. Tom Hertz and I have been examining this phenomenon for 
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several months, and quite oftens^it seems, that there just "'haven' t been'any new 
ideas in a given fie^d.* Or, there haven't been any breakthroughs in instrument 
develo(>ment , no new techniques, and so the field stagnates* Another contributing 
factor has to do with comparability and generalizability ,of findings coming 
out of completed studies* This is our major concern at present. 

In the paper that Tom and I have written, we take the position that very 
often the, problem is at the level of sample descriptiorf and sampling* In 
some cases the samples aren't big enough, .or don't represent populations to which 
one needs to generalize* Very often the problem is Jjast at the level of sample 

4 

description* For example, someone may be trying to do an article for the Psycho- 
logical Review covering the last few years in an active research area* There 
iare many studies, yet the results can't be added up* The main reason is that 
there isn't sufficient information to make it possible to say that the samples 
are similar or in what ways they differ* 

^ Another problem seems to lie in the area of measurement* Very often 
researchers develop their own measures because that's a way to get points for 
being creative, not because the existing ones are inadequate* Then they neglect 
to use any measures that others have used in the past, so that the relationship 
of their work to that of others can be checked* 

The ojther problem lies in the lack of collaboration* Even if there isn't 
a need for large-scale collaboration, some fields could benefit from snail 
scale, more informal, and flexible kinds of collaboration* When people are 
aware of what others are doing they at least have the opportunity to align their 

methodology, instrumentation, or samples do as to assist comparability* 

\ 

Unfortunately, at this time, no matter how one discusses comparability and 
generalizability , the spectre of Government control arises* Although Government 
control isn't advocated in our paper, and despite our best efforts to emphasize 
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that It Isn' t some researchers will Inuned lately see a threat to free research. 
I think there's- an understandable reason for this "knee jerk" response. We're 
all sensitive right now, highly sensitive', to the central regulation of research 
because of the difficult Issues Involved In the protection of human subjects. 
So to write a paper that points to the need for researchers to work tt>gether 
and consider their cumulative Impact Is like throwing gasoline on a flrfe. 
Nonetheless, we think comparability and generallzabtllty ^s an Important area 
that needs to he discussed. 

Central regulation Is not Just a problem of whether there Is Federal -xegula- 

V 

tlon of research^' but whether there Is central direction from any source. It, 
can occur from funding sources within the research community. It can also 
occur because there Is a conformity within the copnunlty of researchers. 
Central regulation can be latent. ^1 can say this to reassure you, that I'm 
sympathetic and equally concerned about central regulation, because in the In- 
/ house research program at NIMH I was also subject to It. The procedures that 

the former Bureau of the Budget hac( In effect tied me up\n*a research project 
for a year. If you had a questionnaire sent out to more than 10 or more jsub- 
Jects. you had to send It up for clearance. It took at least a year trying to 
get a project approved that Earl Shaffer and I were doing on parental attltudea. 
Thus, although I have bee^ in the government for many years befjofe going to the 
University of Virginia, I have had experiences similar to those outside the 
government. 

y - 

As, we looked over the literature for efforts to achieve better doraparability 

/ " ^ ' ^ "^^^ 

and general izability In research, Tom^nd I discovered that these efforts came 

about 8i>oftMneously by investigaj^ors in the field. In my own area of infant 

. research, ^ r^ember Earl Lip ton ^suggested years ago t/hat we fget together to try 

^ to® develop a few measures that most of us could use, no matter how we did our 
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studies. Unfortanately, we never did get together. < »? 

^ V 
% 

Partly as a result of talking to people who haVe served on advisory groups 
to trie Interagency fanels, and consultants, we have dlscov^lred that there have 
been quite a few efforts to do something about comparal>ijLity, spontaneously" 
developed by the -investigators theinselved. >pieir ^experiences are useful becai^se, 
when you knoV what's been done in the past, you can get^ an idea of . potential 
guidelines for the future. These guidelines can be l&sed on what investigators 
themselves have decided needed to be done. In this case we are clfearly not 
talking about something Imposed by a government agency, but what investigators 
themselves feel that T#e need ta do. For example, one of the efforts that we 
meqtloned in^ the paper was the development of the "Upstart" group. The members 
of this group were all involved iti early childhood intervention research, 
efforts to offset deficits ia the environment of children who w^re disadvantaged. 
This group, with very little help, got together and decided on conmon protocols 

for recording sample characteristics. 

, • . ' ' ^ 

Sam Messick tali^%S^ about;^nother group, in this case involved in factorial 

i 

. 8tu4ies of various questionnaire measures of creativity. This group met several 

times, finally agreeing on sets of marker variables that were available to any- 

i 

body carrying out factorial studies in any one of several areas. Kits were 
made available containing measures of these marker variables, making it wery^j 



easy for people to use these measurement instruments. If an investigator were 
planning a study in a certain area of creativity., and he wanted to tap, for 
instance, factors X, T, and Z, there were one or two measures that cbuld be 
included to "mark" those factors. 

Another example comes from the area of sleep research and, again, this was 
a spontaneous development. The investigators Tn this area were conG^erned 
^bout all the different criteria for stages of infant sl<^ep — a very difficult 
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^rea of research. They got. together tcl tackle the proble^n because they ha^^ 
been sensitized by exchanges at meetings which went something like this: "Well, 
I scored variably X this way," and another one said, "I measure it that way," 
and, "I included the transition periods/' and', "Well, IN don't use this measureS 
and I didn't count the transition periods." They found out that. they couldn'^t 
put the. results together. Sleep research had evidently passed through a sta^e • 
where the best thing to do is to just cut everybody loose and let them see 
what they can cpme up with in the way of criteria. Now it was time to get 
together. They developed specimj|^ kits which are used by the members as guide- 
lines. The quality o,f the specimen kits is good because the measures Have been 
worked out by people who are quite knowledgeable in the area. That ia, ,if ^ 
you want to use them, O.K., if you don't, you don\t have to. The individual 
investigator has to make up h^s own mind on how to do Jiis research^ 

Now another point that is brought out in the paper Is that comparability 
and generalizability are very Specific to research in given areas. There are 

areas in which exactly the opposite iis the prob|.em, people have been doinjg the* 

(I 

same' thing for so long that the research has become stereotyped. They have 

been using the satie paradigms, the same experimental procedures, and because 

of this they\ti^not getting anywhere. By way of contrast, very, often thiere is 

an area of research in which very few investigators are involved who are ^ 

teally at the leading edge of a new field, and just getting started. In this 

case it is difficult to see the direction that the reseat^ will take. There 

is no poitit in talking about comparability and generalizability in such areas. 

#■ 

Our main concern is areas in which there has been a lot of research 
and it hasn't added up too well. It may not have added up for many reasons, 
but one thing one can look for is whether there has been enough attention paid 
to comparability and generalizability.' So we Ire very much talking, alx>ut certain 

14 



areas and one kind of feedback that would be very helpful from the participant 
would be: What are the areas in which these concerns are appropriate. In 
which areas are they not pertinent? Comparability and general izability are 
Important to research policy, but n^ on a blanket basis. need to address 
these issues in each individual research area. 
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SUMMARIES OF THE WORKGROUP DISCUSSIONS ' 
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WORKGROUP I 

♦ 
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University 

i 
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WORKGROUP I 
Simmaty of the Discussion 

• Moderator : David Pearl H 

National Institute of Mental Health 

Before they ditfcussed comparability in research and the accumulation of 
scientific knowledge, the participants in this workgroup focused on the current 
climate foj research in the nation. There was agreement that Congress and 
"the public are becoming increasingly skeptical about research activities and 
are demanding greater accountability in research. According to some partici- 
pants, this growing disillusionment is aggravated by a general failure of scientists 
to communicate' the outcome of their Research to these non-scientific audiences. 
\ While researchers are usually trained well to write articles for scientific ^ 
Journals, most are neither able nor willing to writte reports that are both clear 
and compelling to non-scientists. Steps in this direction will have to be under- 
taken, however, if the climate for research is to Improve. Participants pointed 

out that without adequate funds for research, significant progress in solving 

A 

scientific and social problems will be difficult to achieve, no matter what 
* ' courses might be charted for. the training of researchers and the practice of 
research. , ^ 

Can researchers themselves be expected to report their findings clearly, 
promptly and effectively to the general public? The feeling among the discussants 
^ was that researchers should not be asked to take on this additional task. One 
suggestion which received considerable attention was that more science writers 
be trained to meet this need for liaison between the scientific and non-scientific 
comunities. A few small pilot programs might be designed to train individuals 
to synthesize research findings, and then "translate" this scientj.fic informa- 
tion into everyday language. ' * 

The discussion turned next to the need for improved dissemination of research 
findings and exchange of information within the research community proper. Many 

18 
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of the participants expressed dissatisfaction in particular with current policies 
of research support which allow unpublishe.d technical reports to be buried' in 
agency filing cabinets, and long periods of time to elapse before the proceedings 
or reports of Federal meetings iire made available to the research community. 

Noting that proBlems associated with the process of reporting tindings were 
of only peripheral importance if the findings themselves could not be meaning- 
fully analyzed in the first place, the discussants shifted their attention from 
the issue of dissemination to the primary focus of the conference, comparability - 

4 

in research. There was general agreement that diversity io concepts, measures 
and methods often made it impossible to * synthesize research findings from dif- 
ferent studies e^en within a narrow sub-area of research. Participants complained 
that many areas are placed by confusion because of differences in the way 
researchers define their terminology. Whd[.l& there are certainly newly develop- 
ing areas where effdrts to decelerate agreement on concepts iftight be premature^ 
there are countless areas where agreement on definitional matters would be feasi-^ 
ble and fruitful. Participants endorsed the suggestion that Federal agencies 
support more conferences or workshops where investigators working within a commpn 
substantive area could convene ip order to exchange information and try to develop 
a conqnon frame of reference. 

In addressing themselves to specific mechanism^ for increasing comp\ra- 
bility, the discussants agreed that standardization of measures and techniques 
could be beneficial to a field, but warned that it should not be carried out to 
such an extent or in such an inflexible manner, that it routinizes research and 

J 

discourages new theoretical developments. Comparability should be promoted so 
as not to discourage the continuous questioning of the concepts, measures and 
methods uded in an area. This caution was thought to be particularly germane 
to the use of marker variables. The process of selecting specific marker 

19 
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variables and measures, even within a narrowly circumscribed area of research, 
will be extremely difficult. Discussants argued that we will never reach the ^ 
point where all researchers will be able to agree on which marker variables to 
measure, in any one study, much less in a number of studies across an entire 
area, and over an indeterminate timfe period. For one things ,the appropriateness 
of a variable as a marker will derive in part from the. particular theoretical 
approach favored by ^,the researcher. Accordingly, the participants opposed any 
system of directly regulating comparability, an(} especially one that would Impose^ 
a specific set of vai^iables upon an ^rea. On the other hand, while they felt 
that it was inadvisable to force investigators to adopt specific measures, they 
thought that it was reasonable to ask investigators to make an effort, in their 
own fashion,^ to clarify the relationship between their research terms, measures 
and techniques and those of other research projects which focus on similar 
problems. Specifically, researchers might be encouraged to select their own 
marker variables and meas\ires, or to modify^ those selected by ot-hers* as they* 
deem It necessary. In this way, marker measures could be used to increase com- 
parability within an *area of research, yet because these measures would be 
altered or replaced as they are jAidged obsolete, irrelevant, or simply less 
informative than other measures, the area would continue to evolve freely, 
without undue standardization. Furthermore, even if it were not objectionable, 

V 

on the groundd outlined above, to give the responsibility to a small group of 
experts, the task of^determlning the mdst effective measut^es to increase com- 
parablllty in a colitinually changing area of research, would be inordinately 
difficult and time-consuming. The task might be more feasible, however, if 
it were left to the individual researcher to decid^which marker measures to 
include. By explaining his reasons for selecting certain measures rather than 
others, each researcher would be contributing to a general theory of comparability 
and marker variables within that area. 
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Participants pointed out that as an area of research evolves, so too do 
^ the needs for information. A variable, 'dsuch as eye color, may seem irrelevant 
to a particular research problem today, but turn 'out to be significant 
- tomorrow/ Can investigators be expected to predicjt future needs when they design 
• their studies? One solution might be to overrreport background information. This 
, .Suggestion prompted some debate among the discussants concerning the extent to 
which an investigator could be expected to include "extra" information or 
variables beyond those that have immediate relevance. Some participants thought 
that over-reporting was preferable to under-reporting, and expressed their own 
willingness to collect any additional information, as long as they wer 6 funded 
accordingly, and the means for storing' the resulting data were provided. Others 
said that they would not be willing to include variables which did not appear 
. to have any real bearing on the processes being studied, and contended that the 
collection of information on a vast array of variables, 3elected on the chance 
that they may prove to be relevant someday, would impose an unreasonable burden 
on the researcher and constituted i>oor scientific procedure . The number of 
potentially relevant variables, it was argued, would be so large that this 
strategy would not be practical. ^ * ' 

With regard to the question of how to store extensive background informa- 
tion, roost participants took the position that there was no room in journal 
articles to include data which were not directly relevant to the study being 
reported, and that it was too costly and cumbersome for researchers themselves 
to store comprehen^ve background data, especially over long periods of time. 
The possibility of establishing data banks for marker variable^was sugge^ed. 

A theme that recurred throughout the discussion was that there are many 

J 

institutional arrangements and pressures within the research community which 
inadvertently discourage comparability in research and' the synthesis of research 
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findings. Many kinds of research activities which are critical to the accumula- 
'tion of scientific knowledge, such as reviews, replication studies, collabora- 
tive research, methodological research, and instrument development, h^ve relatively 
little "pay-off" in terms of institutional rewards. At the tline thei are being 
trained, graduate students are pressed to be innovative and to undertake work ^ 
on new research problems, not to replicate prior studies oi: build directly on 
the work of others. After graduate training is completed, the researcher is 
faced with new pressures which reinforce the same basic pattern. Since tenure 
decisions often are based more on the quantity of work accomplished than the 
quality or significance of the contribution, the system tends to discourage long- 
term, sequential, or collaborative studies. . * 
According tcuone participant, developmental psychologists have been' delin^ ^ . 
quent when it comes to validating tjheir own instruments. In contrast to the 
early years of research on children, there is little interest in scaling, and 
too seldom do researchers carefully examine the properties of their tests and 
measures'^, even when they administer them to children in different age groups. 

Toward the end of the session, the participants considered specific mechan- 
isms that might be used to attain greater comparability in research. A coopera- 
tive research model employed by the USDA "was described to the group. The USDA 
often brings together investigators from several universities who are interested 
in working on a common problem, and helps them plan their objectives. With 
this approach, data which is collected in different states or regions of the 
country can be meaningfully compared and sometimes even pooled. An interesting 
aspect of the strategy is that it allows for both comparability and flexibility. 
While the different groups may agree on certain aspects of design, methodology 
and sampling, a group might also pursue its own interests and apply its own 
l^nd of expertise in particular areas. 
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Returning to the topic of marker variables, the discussants concluded that 
the moat appropriate way for the federal agencies to Increase comparability In 
research is not through regulations or guidelines concerning the use of marker 
variables, but through support fpr conferences in which researchers working 
within common substantive areas could .clarify and synthesize theoretical con- 
cepts » measures » and methods involved in their research. An agency representa- 
tive polnted^out that some Federal agencies are in fact responsive to suggestions 
from researchers for conferences in areas of concern to them. Participants 
complained that it was difficult for many researfchers *to find out what mecha^nisms 
already are available, and it wa3 recommended that channels of communication . 



between agencies and rfesearchers he Impjroved. * " 

'1 ^ ■ • . .,, • ■ 

Literature reviews and state-a^-the-art papers were endorsed as other means 

■ ■ ■ ^ ■ . ^ : 

of making research more cumulat;Lve. Sotae participants cautioned, however, that 
an aimless explosion of such iictiv^ties would be wasteful and. unproductive, and 
that the need for suet papers should be carefully weighed within each particular 
area of research. 

Strong support was expressed fpr aixy steps that would Improve the researcher's 
access to scientific reports and data. Several participants suggested the need 
for a computerized information atolrage and retrieval system which would be 
comprehensive (e.g., provide access to unpublished government reports and biblio-c 
graphies as well as published dociinents) , and reaHily available (and inexpensive) 
to the Individual researcher. The better acquainted researchers are with the 
latest developments throughout their areas of interest^ the easier it will be 
for them to build some potential for comparability into their research. 

With regard to graduate training programs, participants saw need for change, 
but were not optimistic about the chances of altering practices which run counter 
to comparability and the' synthesis of research findings. As one participant put 
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it, we train graduate students primarily to carry out research that will be 
publishable» and until the current system of rewards is changed, it is not 
realistic to expect graduate programs to change significantly. Nevertheless, 
it was argued that greater attention should be given by universities to the 
kind of problems raised in this conference, as they relate to training programs. 

Finally, there was some discussion about the source of funds for activitijes 
intended to increase comparability in research. Unless new appropriations are 
sought, which most participants thought unlikely, any, extra funds required will 
have to be diverted from some other areas or types of research activity. Several 
discussants . stressed that decisions to Implement efforts to increase co^ara- 
bilitycould not be made in the abstract, *and that, budgetary realitfe^ would 
have to be taken Into consideration. 
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WORKGROUP II 
Svflnmary of clie Discussion 



Moderator : Richard Q. Bell 

University of Virginia 



As the session began, participants acknowledged that there was a definite 
need for Increased comparability In i;esearch. Providing examples from their 
own areas of Interest, they pointed out that the continual development of new 
concepts, definitions, measures, and techniques makes It difficult to synthesize 
findings from different studies. There was considerable give and take con- 
cerning the means by which this situation might be l^nproved. General support 
was expressed for. the use of marker variables as a strategy of Increasing com- 
parability. Participants thought that, given appri6prlate funding, many 
researchers might be willing to include background measures that would relate 
their studies to other studies focused on the same generad problems, even if 
these^ measures were irrelevant to their owti hypotheses. 

Although the marker variable strategy was not questioned, some felt that 
the attual task of selecting demographic and marker variables could present a 
dilemma for researchers. Ideally, in order for researchers to select the pa?:- 
tlcular variables that will best enable research studies to be tied together, 
they would have to identify variables that would be relevant Jto future concep- 
tualizations of the research problem. After all, as most areas of research 
have evolved investigators have discovered that, in order to understand the 
processes being studied, they have had to consider factors which previously had 
' been ignored completely. For instance, in many areas of researcTi, data from 
subjects of both sexes has always been pooled. Investigators then discover 
that there were in fact significant sex differences in performance on experi- 
mental tasks. Clearly there is no way for researchers to foresee future devel- 
opments. One way to help researchers select the most effective marker variables 
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might be to enable them to meet periodically to assess progress In a particular 
area of research, and to Identify new directions or promising approaches and 
measures* 

Along the. same lines, participants pointed out that In some areas dif- 
ferences in the way researchers conceptualized their problems and defined their 
tenas were so great th^j^^^y attempt to Increase comparability through the use 
of marker variables wotu^be premature. The first priority In such areas 
should be to bring researchers together to clear up the conceptual "underbrush". 

Questions about the role of the Federal agencies In efforts « to Increase ^ . 
comparability prompted discussion about priorities In research support. ;Par- 
tlcipants wondered whether agencies \^uld encourage Increased comparability 
across the l^'oard, or would concentrate on a few high priority areas. .Similarly, 
there was some debate as to whether the use of marker variables would be encour- 
aged primarily ii^-research that was applied or had social policy implications. 
Although one participant contended that in evaluating large scale intervention" 
programs it was especially crucial to be able to compare different studies, the 
group concluded tra|t there? was a need for comparability in basic research as 
well as applied research. 

c ^ 

While on the topic of agency priorities and pland,^ discussants suggested 
that fra^entatlon of research effort stems in part from the researcher's lack 
of information about agency priorities, and recommended more effective coynnunl- 
cati^on of agency goals and Interests to researchers in the field. Another aspect 
of. this topic concerned the manner in which agencies formulated and-^pursued 
their research goals. The present system, participants suggested, discouraged 
comprehensive and cohesive programs of^research. It was argued that grants 
were awarded most often to investigators who undertake new and different kinds ' 
^ of researcht as opposed to replications, cooperative projects, or studies that 
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tle in closely with prior work or other ongoing work. Some participants hypo- 
theslzed that a systems approach to research might be the logical implication 
of Federal interest in comparability and cumulative research. With this 
approach, more^ommonly applied to the physical sciences, proble|^ most in 
need of solution would be defined,- and ^ or each problem possible hypotheses would 



be identified. Different groups would systematically attack these hypotheses, 
/ proceeding to others as ^ome were eliminated. It was commented that this 
model of research had not been as successful as had been hoped on the few 
occasions that it had been applied to the social sciences. No consensus was 
reached on the advantages or "disadvan^tages of this kind of coordinated research. 

The group listened to a description of a model of regional ^nd national 
planning used by ^he USDA, wherein researchfers and administrators^ from the 
agricultural experiment stations get together to examine problems in agri^culture 
and rural lifie, to evaluate research anfi the knowledge it produces, and to 
point out directions for future' research. The USDA often uses panels of 
investigators, who ar^ themselves dra\m from the experiment stations^ to 
review the program of a particular experiment station. 

The description of the USDA panel-review system sparked a discussion 
about the feasibility of instituting a system of panels that would address 
pressing problans in early childhood and adolescence research. The. following 
general proposal for such a system emerged from the various comments and 
recommendations. The panels co^ld be comprised of experts in the field, who 
could monitor particular areas of research on children and youth in order to 
evaltiate research progress, identify problems In need of attention, and 
recimmend how to distribute efforts in order to increase comparability and to 
make research more cunulative. The primary function of the panels would be 
to help researchers come up with answers in areas in which there is pressing. 
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public need. One primary task %w>uld be to clear up any'^^xfonf usion caused by 
differences between .incepts, definitions, and measures used in particular 
areas. It would be the panel's responsibility to see that all relevant lines 
of research were showing progress, whether this involved basic res^rch, 
applied research, conferences, or reviews and syntheses of the literature. 
Thus the paiflH would help to fill in the kind of gaps in our present research 
system that have "^een identified in both of the Interagency Panels' conferences 
on ^comparability . Presently, this function of adding up the pattern of research 
is "falling between the tracks", with no one element of the research community 
capable of taking it on. For instance, editors of journals cannot be asked to 
play this role; they do not have access to all studies in an area, and, at any 
rate, the task of assessing comparability and progress in a general area would 
be overwhelmingly time-consuming for a small group of individuals. By con- 
centrating on the overall picture of research and the long range dLmplications 
of work in an area, the panels could provide researchers with information and 
help that is not available from other sources, such as the researcher's col- 
leagues. 

The panels might be supported by foundations and Federal agencies, but 
it was suggested that the members might bfRpppointed independently of these 
institutions, perhaps by professional societies. Members should be drawn 
from many disciplines, and from all sectors of the research community, includ- 
ing the Federal agencies. Vhlle the membership of the panels could be rotated 
in order to assure the steady influx of fresh ideas, terms could be staggered 
to provide continuity in the panels' activities. One participant cautioned 
that with the broad interdisciplinary scope of the panels, special efforts 
would have to be made to keep the panels task oriented and to prevent them from 
turning into Ineffective bureaucnracies. The discussants considered the question 

/ 
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as to whether such panels might tend to become conservative and have a 
dampening effect on the development of novel Ideas. Most agreed that the 
panel approach might encourage conformity to some extent, as does any form of 
peer review, but since the panels' role would be to advise and not coerce 
researchers, the discussants concluded that the advantages would outwelgti the 
disadvantages. 

The next topic considered by the group was graduate training. How can 
graduate programs more effectively sensitize young researchers to the prob- 
lems of comparability and cumulablve research? It was reconmended that more 
opportunities be created for graduate students to spend a yeair as Interns In 
Federal research agencies. In order to learn about the whole range of problems 
and Issues that relate to the synthesis of research at the national level, 
and to the Interface between research and policy. 

A major problem, according to participants. Is that the topics of com- 
' / 
parabillty and the selection of measures simply receive short shrift In most 

graduate programs. The ability to design research that ±d cumulative derives 
hot so much from specific technical skills. It was felt, as from a general 
sensitivity to what Is going on In the field as a whole. Even If specific 
methods to Increase comparahlllty cannot tai;ght directly, however, at least 
a better feel for probl^s Involved In synthesizing and comparing studies could 
be Imparted to students through courses and textbooks. A good way to help 
students become more sophisticated In the^ selection of marker variables might 
be to write an historical textbook that specializes in "critical incidents" in 
which t;li^ course of research in various^^reas was altered when a previously 
ignored variable was discovered to be relevant to the processes under investi- 
gation. Also, reports analyzing the comparability of measures in specific 
areas of research, such as the recent doctment on family research prepared for 
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the Interagency Panels, would be suitable material for graduate courses. 

Discussants noted some apparent trends in graduate training which might 
increase sensitivity to problems of comparability. Perhaps most significant 
is the substitution of review articles for the more traditional area examina- 
tions, as the qualification for Ph.D. candidacy. This approach may help the 
student to see the relationship of individual research efforts to a larger 
pattern ot long-term efforts. The group recommended a thqrough examination 
of the function of the key graduate requirements, such as the qualifying 
examination and the dissertation. For instance, should students be expected 
to make a unique contribution in graduate level research? Or should they try 
to fill in a gap within an area of research, even if that were to entail a 
simple replication? 

Another area of concern pertained to the graduate student's access to 
information. To some extent, the graduate student's knowledge of what is going 
on in an area of research is contingent on the expertise and involvement of 
the faculty at that institution. Without access to preprints and other 
materials exchanged through the informal but often comprehensive communication 
networks that exist between many researchers, students must depend primarily 

A 

on published materials, which report research completed much earlier. The 
discussants emphasized the need to Improve the dissemination of unpublished 
information, including technical reports deposited in agency files, in order 
to enable graduate sjtudents (and all researchers) to study the field in a more 
systematic way. Similarly, data banks should be open and available to interested 
researchers. Complaints were expressed that some resea;?lhers guard their data 
too ^Jj^lously, refusing, even after a reasonable length of time, to share it - 
with other interested researchers who could carry out further analyses. 



With regard to the general climate fpr research within univfet^ityydepart 
ments, participants argued that there is a need to modify the existing reward 

i 

system. The system by which tenure is granted, with itri ejnphasis dn the 
quantity of an individual's research, discourages progranmatic and long-term 
work. Similarly, cooperative research carried out by several investigators 
generally carries less prestige than do studies with single authorship. 
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WORKGROUP XII " 
Sumnary of the Discussion ^ 

Moderator : Carol McHale 

DHEW/Office of the Assistant Secretary 
for Planning and Evaluation 

The keynote of disciission in this group highlighted interdependent 
issues to be addressed: 

1. Is comparability in research results a desirable goal 

and to what degree has it been achieved in research on . 

early childhood and adolescence? 
t 

2. Given that there is a problem, can government and the 
research community work together toward a solution? 

3. ^ What are appropriate strategies for doing this? 

4. What are appropriate roles for the government and the 
research community for accomplishing these strategies? 

Immediately a question arose as to the goals facilitated* by comparability 
in research. On the one hand will greater comparability allow for mor^e 
sophisticated conceptual systems, or on the other, more effective social 
action? The group felt that the use of comparability approaches in research 
toward applied soci£^l policy is more appropriate than that directed toward the 
basic conceptual systems* When efforts are directed toward development of 
basic conceptual systems a certain amount of error is expected and tolerated. 
The same error in development and implementation in applied social policy 
can result in the loss of huge amounts of money and severe damage to the 
reputations of the social sciences in the mind of the public. Later on in the 
group session the developing thought was that comparability related to research 
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directed to applications is a more appropriate role than in* development of 
basic conceptual systems especially in the initial .stages. 

The main concern voiced in th^ opening session of this conference was: 
how do the social sciences answer to the policy makers in terms of research 
output and research provided answers. The agency directors, according to 
one participant, are not interested i^ comparability per se, rather they 
ask, "Why don't you have any answers?" And the reason that researchers don't 
have answers is that the answers are so disparate, so different as to results 
concerning the same phenomenon that we are reluctant to make a definitive 
statement. Another participant asked if It would be possible to get some sort 
of survey on the number and type of questions asked by the policy makers in 
the legislative and executive branches of government. 

One suggested response to this question Is to gather together the questions 
policy makers ask, probably 5 or 6 basic questions which keep recurring. Then 
gather five or six of the best people you can find on a given issue with five 
or six consumers of information on that same issue and map the research needed 
for that issue. The answers which emerge stand the best chance of being the 
right ones. 

The group turned their §^tention to exploring ways to achieve comparability 
and the subject of marker variables arose. In the dourse of the discussion, 
the groiip f^^ that in using marker variables to achieve comparability in basic 
conceptual systems certain problems emerge. Using as an example the early work 
on stuttering, one participant recounted that some 30 theories were advanced, 
most have since fallen by the wayside as research results refuted them. To 
try to establish definitions and marker variables during this early multi-theory 
period would be extremely difficult if not futile. The group consensus was 
that there is a developmental period in a research area during which not enough 
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ls known to specify marker variables to deo^^op conmon definitions. As 
fields develop they may become "ripe" for movement toward comparability. 
Short term men\ory was suggested as a field which might be appropriate for 
marker variable inclusion. The characteristics which support this contention 

/ 

are: (1) large number of investigators; (2) relatively similar methodologies^ 
and procedures; and (3) a fairly well circumscribed area of investigation. 

The chairperson opened a possible area of profitable inquiry, that of 
process, posing tU^ question: What steps would facilitate decisions on marker 
variables in the research process? 

One basic condition preliminary to the use of marker variables is suf- 
ficient knowledge of the phenomenon under study. Without this knowledge no 
intelligent choice of functional marker variables can be made. One participant 
observed that the selection of marker variables is allied to predicting the 
future. One must select those variables which are pertinent to the relating 
of a present study with a future one. A number of participants voiced the 
opinion that the people working in a given area are thS ones who are best 
qualified to propose the variables of significance for marker variables. This 
concern was voiced in the light of mandatory specified marker variables and 
other dictated procedures sunmed up in the statement: If comparability meanSs 
prescription or proscription, it is not a good idea. The Panel members present 
vehemently responded that this was not the intention of the agencies. 

One suggestion was in describing behiaviors and procedures in a given study 
to provide benchmarks for comparability. Another of the participants remarked 
that it is possible to get significant results with extres^ly small samples 
provided the procedures and behaviors are carefully described. He used the 
work of Piaget and Roger Brown as examples and ventured that these small sample 
studies produced more knowledge of the structure of language than all of the 
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hundreds of others that have been done in the area. The major reason is the 
length of time devofied by those two researchers. To gain Insight, and under- 
standing the researcher has to stay with it over a period of years. Others 
cannot replicate the changing conditions nor achieve ecological validity, but 
with careful study of the interaction of variables and because the conceptual 
framework is clearly explained and understood, comparison is possible. Unfor- 
tunately, the present Federal funding patterns tend to preclude this with the * 
usual one year study or at most the three year study. 

The mention of Federal funding patterns gave rise to a new topic. In 
achieving greater comparatility and generalizal^ility in resterch a number of 
problems can be anticipated and. from a variety of sources. The patticiparfts in 
tM group pointed some of these out. One problem is that of Federal funding 
patterns. ^The analogy was made to a "pie" in which 80Z may be devoted to 
directed research and 20Z to non-directed, but the entire pie will shrink as 
long as policy makers see non-ed:|,bles emerging from the pie in terms of non- 
generalizable research results or contradicting ones. In the same fiscal 
vein, there is a problem witl;^ marker variables in the case where the marker 
variable and its associated measures are so complex and expensive to collect 
that the main research effort is adversely affected. 

The participants turned their discussion to the question of replication and 
the inherent' difficulties. In connection with these dif f icultires, the 
importance of the "Joe Mqran" factor was mentioned. This factor was concerned 
with the unique input of unmeasured variables such as afl'^tstanding teacher 
(i.e., "Joe MEitan") which tend ^o be Ir reproducible in replicative studies. 
Replication and comparability also tend to be based on the organizational 
structure as well as a time sequence. This organizational structure is 
extres&ely difficult to reproduce and to explain to others for the purposes of 
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comparability. Two reasons why comparability is weak are: (1) researchers 
do not know how to achieve it, or (2) they've forgotten. The first can be 
ameliorated by graduate programs, the second by "reinforcement of the reinforcers" 
so that the researcher/professors are reminded of the need for comparability 
as well. 

All in all, the group felt that comparability could assist in |^&9*ering 
replication of significant studies to test reliability' and validity of the 
findings, which is a hallmark of good science whether it be applied or basic 
research. 

Toward the latter part of the session, the group began to explore ways to 

t 

resolve some of the problems the issue of comparability poses. Some of the 

suggestions follow. 

• -One way to enhance comparability and generalizibility in research 
would be to encourage more articles on the subject in order to call 
attention to it, especially in the training of graduate students 
and to remind the established researchers to attend to the same 
problem. 

The second proposal stated that a multivariate approach to marker 
variables «hould be investigated. A thorough multivariate analysis 
would identify the best estimates as tp whether or not- marker variables 
are valid and include, measures of them to provide more definition as to 

\ 

those that are truly valid. Those areas of study which can accommodate 
multivariate analysis are potential ones to examine for marker variable 
applications. ^ 

The third proposal indicated that although comparability in this con- 
ference has explicitly been used to relate research to research. Implicitly 
there is the need for cofiiparability from research to development to 
evaluation to dissemination to service. 
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It was pointed out Jthat translating the conceptual framework of a 
theoretical system into practical applications is a different type 
of task than the initial intellectual effort in developing the original 
framework. It can be facilitated by a highly knowledgeable group of 
5 or 6 specialists sitting down xo get to delineate the best potential 
approaches to the "problem and then coordinate by their working closely 
together using a comnoQ set of variables over 5 to 7 years* In 
developing comparability in research a number of participants indicated 
that directed research on a comparative basis may have to be initiated 
perhaps on the USDA model of cooperative research. 

Research In the area of Instrument development seems to be an impor- 
tant component in furthering comparability via marker variables. For 
an effective marker variable approach, there has to be reliable, valid 
instrumentation in the research area and too many areas lack this 
quality of instrumentation. 

Some consideration was given to the '^natural" development of a research 
area and how this relates to cbmparability . In fact, some participants 
felt that "auto-comparability" will occur as a field of study becomes 
better understood. It tends to be an outcome, but whether it can be 
Incorporated as an input factor to hasten the arrival of the "well 
understood" condition is unknown. It was suggested that study be 
devoted to this possibility. In this context, the developfilent of 
definitions in a field is seen as an aid to aligning results. Defini- 
tions tend to clarify distinctions. 

*'Maxlm: Comparability emerges in a field when the researchers discover 
what functional variables are Important to the field." 
a closing note the group decided with regard to marker variables as 
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an approach to comparability, the field has got to arrive at the point where 
enough is known to determine those variables which are "functioning variables*" 
There is a tremendous amount of social pressure to include those variables 
which are recognized as having a potentially important Impact on the phenomenon 
under examination. This tends eventually to form a self-correcting mechanism 
via critiques of the reports generated by research in particular areas, but it 
must also ,be ex post facto , perhaps 3 to 5 years based on the publication time 
lag. When the field is aware of the importance of a particular variable there 
will be a gort of automatic comparability as people fall into the method of 
choice that becomes clear in a field. Until that happens it may be dysfunctional 
to push marker variables and limit the researcher's options to discard non-' 
functional variables by institutionalizing them. 
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Planning and Evaluation 

Jualynne Dodson, Atlanta University^ 

Kathleen Hoover Dunlop, George Peabody College for Teachers 
Joseph Click, City University of New, York ^ 

- 

Elizabeth Hagen, Columbia University ^ 

Maure Hurt, Jr., Social Research Group, The Geprge Washington 
University \ 

Anita King, National Institute of Mental Health 

Levis P. Lipsitt, Bro%ii^^ University 



John W. Loughary, University of Oregon 
Martin Manosev^tz, University of Texas 



Halsey M. Marsden, National institute of. Neurological and 
Communicative Disorders and Stroke 

Julie Phillips, National Institute of Mental Health 

^erbert Simons, University of California, Berkeley 

Jerome Taylor, University of Pittsburgh 
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WORKGROUP IV 
Summary of the Discussion 

Moderator : Joseph M, Bobbitt 

National Institute of Child Health 
. and Human Development 



If comparability is vital in research, that is, if the results of research 
study must be potentially comparable with results of other studies or problems, 
how can this be achieved? The group was asked to address this concern in the 
context of training the future researchers and from their own perspectives as 
researchers. 

In the general discussion that followed, one Immediate response „was that 
for the graduate student, in the development of the conceptual framework of 
the dissertation, comparability is vital. On one side of the issue, the student, 
mu^ have the skills necessary to make the comparisons and then synthesize. On 
the other, the research should be so organized, and executed that the potential^ 
for comparability does actually exist. 

A whole r/ing^ of problems hinders comparability. Comparability based on 
the use of instruments and methodology tends to suffer over time as new develop- 
ments or Improvements come about and instruments and methodology become out- 
moded. This is "instrument decay," especially critical in long term longitudi- 
nal studies. Actually, the fact that we have a problem of comparability with 
older, studies indicates progress in itself, as was ruefully observed by one 
participant. ^ 

Comparability is an Issue of differing magnitude in different disciplines. 
It was mentioned that additivity of research results is very important in 
sociology whereas In some areas of psychology it is not seen as a highly 
critical issue. One of the participants related that anthropology has long 
attempted to accomplish inter-cultural comparisons from an external viewpoint. 
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now there seems to be a trend to examine cjfitures from|an Internal one. Some 
of CCD's family studies projects relate to this point. Even though external 
demographic variables may not aid substantially. If the Investigators could 
collaborate, a more "Internal" focus would be Initiated. The structures of the 
projects could be more closely correlat ^ ^through facilitating comparability. 
In line with this thought, some of xhe participants felt that the return would 
not be worth a great effort to develop a return on the dapdgraphlc level marker 
variables. They do not proceed that far along the road to comparability and 
generallzablllty that the research Is seeking. 

As a hypothesis, there m^y be different leve^p of comparability, one Is 
on a superficial/demographic level, while the other deeper one Is on a con- 
ceptual level. This deeper level was perceived as having a greater payoff, although 
much more difficult to achieve. 

-The discussion then switched to the process of research and research 
funding and how this might affect comparability. This need for comparability 
Is observed at policy and legislative levels. One of the Panel members observed 
that the results of applied research are often needed to answer congressional 
Inquiries and that from this vantage point comparability Is Indeed critical. 
Unfortunately the means to determine comparability ar^scarce on the descriptive 
level and almost nonexistent on the conceptual level. In moving to address 
this need among others, a Panel member noted that the basic funding mechanism 
of the Federal government has been the grant and still Is, but as the current 
trend seems to Indicate, the contract i^^^^ecome more and more prevalent. 
.This means the directions and condltij^ip^^^ will be constrained by the 

requirements set forth In the RFP (Reqiii^dt for Proposal) and, although the 
government scientists and research administrators strive to do it right, it 
remains a difficult task. In response a participant noted that if a marker 
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variable approach Is adopted In totQ ,to require the ^vrong" marker variables 
would b€i a terrible mistake and even the "right" ones could have negative side 
effects- The major danger ^seen is in concentrating on the tabulation of marker 
variables at the expedse of the^eal purpose of the investigation. 

On thie other hand, it was countered, some of the proposals funded by 
the Panel agencies are lacking in even the 4emographic level variables which 
may be superficial In some ways, but still this lack demonstrates a lack of 
sensitivity to the problem. 

Following up on the negative side effects, an example of misuse of marker 
variables was given' concerning .a mldwestem university which required a specific 
standardized test for subjects in all doctoral studies. These tests were 
normally administered by the school districts and the graduate students would 
get the data from school records. Unfortunately, those Children in districts 
which didn't use the test were excluded from any studies and, unless the doc- 
toral students had the time and funds to do their own test administration, the 
results were unavailable* 

When specific marker variables were discussed, one that received attention 
was socioeconomic class. SES is related^ to a wide range of social behavior 
and differentiates between leVels. SES as a marker variable is a very complex 
variable with perhaps too simple a marker meastnre. The functional level of 
this concept is extremely powerful so that even peripheral measures' have high 
correlations. The complexity of the construct shouldn't be judged by the 

measures used. In fact, the need for research on the variable may be put off 

\ 

0 

by the ease of scoring at least a part of SES with quite simple measures, such 
as income* If there could be a satisfactory form of meltsurement it would cer- 
tainly help explain inters tudy differences in results. 
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lo effectively use ma variables for comparison of results, their 
"moderator" effect must be known. That Is to sa"y, the relationship^ between 
the phenomenon under Investigation and the marker variable must be known^ 
together with any potential interactions. This level of knowledge Is directly 
related to th^ threshold level of development In a field. The Panel members 
seen to be In a strategic position to recognise research areas that are at a 
critical stage of development and "ready" for coprdlnatlve meetings. 

A suggestion was made to develop "^a "model" of an area of Inquiry and see 
what descriptors or marker variables ''could be applied and then test the 
potential for establishing relationships and the functional ability of that 
particular set of marker variables. Perhaps the set could be changed and 
retested. Then If It seems feasible » formulate the field approach and require- 
ments. There could be two levfela of marker variables; one at the descriptor 
level and a second at the conceptual level. The second level Is difficult to 
reach when the theoretical framework of the field Is not well developed. 

A pg^irtlclpant noted that one part of the marker variable approach which i 

may be misleading Is the concentration on the Individual. When conditions of 

measurement are varied so are scores. This would Indicate that we should look 

at measurements of environment as critical to comparability as well as measurement 

t 

of subjects. Marker yarlj 1}les iLn the ecological setting of the experiment such 
as sex of experimenter » y^^r could be Important a s" marker variables 

too. \ 

The Panel members voiced an overall request for definitions of what should 
be supported In the area of comparability and then asked the University par- 
tlclpants for backing In such a proposition. 

In response, one participant said that the Idea Is, that while we as 
researchers endorse the general move toward greater comparability lif research, 
and we endorse the Idea of pinpointing! tagging, and highlighting Individual 
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difference variables, such as are Implied by socioeconomic status, the age and 
grade level of the Children that are being studied, the race of the subjects 
and so on, we 4lso have reservations ai^d apprehenslons^bout the trend In that 
direction If appropriate duallf Icatli^ns and If appropriate correctives arie not 
made. One <^ the very most Important correctives would be to Introduce additional 
monies and addltjlonal orientation toward the study of these^very Individual 
difference var'^^^les that we want to have highlighted, that we want to use as 
markers. There has to be an Investment In the study of Individual differences 
among the subjects whose behavior or other attributed are explored, and there 
has to be further study of the technology, the tests, or whatever the measures 
are, that are used In these studies. So that, for example, although the cover 
sheet of an application might say the WISC, Is going to be used, there should 
also be some Investment on the part of the researcher and on the part of the 
agency that's going to support the research to further explore the natijre of 
that Wechsler test, and the ways In which Individual difference attributes come 
Into play In performance on that test. 

As the discussion turned to the actual use^of marker Variables In research 
and the groups talked about when and how marker variables could be used, the gr6^ 
felt that there seems to be a threshold level In the development of a field 
before It Is intellectually profitable to sit down together to delineate the 
conmonalltles In a research field. The researchers worklhg on the psychophysiology 
of ^leep were ready to do that and were able to develop comi^n definitions and 
commonly understood methodological approaches. 

Meetings of researchers have been effective In bringing some cxoss-^f ertlll- 
zatlon of techniques and methodology. A qupbtlon arises as to how well this 
means can operate at the conceptikal level and with theoretical Issues. Too 
heavy a concentration on marker variables can have^ toifortunate consequences In 
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- that more problems will arise than are solved. The over concentration on 
methodology will obscure the substantive questions, especially in the absence 
of a theoretical or conceptual framework. There is a problem in the instance 
where camparal)ility beccimes of. such paramount concern that creativity is stifled. 
" ^ Marker variables and comparability should not be^seeri as an end in itself, but 

as a facilitating mechanism toward achieving more productive consequences from 
research. ^ 

The starting level for comparability in the more applied research levels 
or in evaluative research should probably be at the local level and h^ild upward 
rather than at national level and Be reflected downward. In this, the group 
felt that it would be helpful in a dissemination effort to have the Social 
Research Group supply papers for distribution to the participants of seminars 
held at and by the universities for graduate students and faculty. This help 
would be even more appreciated if a staff member or Panel member could act as 
a resource person at regional meetings hosted by a university. 

Although the participants felt that long-term change could be achieved 
via graduate school training, a more Imnediate Impact would be seen if the 
active researchers were addressed. It was noted that very few of the graduating 
Ph.D. s will become active researchers. ^ 

There was a feeling among the university representatives that funds should 
be available to coordinate research and to involve the researchers themselves 
•in this coordination. In this effort, "face to face" coordination should take 
place. In conjunction with this, data exchange would be a good thing. Let 
-X one investigator have a small grant to analyze the data of a second investigator 
from a different perspective and better yet, get the two of them together in 
direct contact to discuss the differences and similarities within their respective 
analytical frameworks. One drawback to the exchange of data especially when the 
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work is done at different points in time, is that extensive record keep4>ng of 
data becomes not only time consuming, but is affected by other factors such as 
invasion of privacy concerns and the resultant safeguarding and constraints on 
release of sensitive information. As another source of data for comparability 
estimates, the final reports of Federally funded projects are more likely to 
have information- for comparability t?han the information in the literature. 
These reports should be in a repository and available to the research community. 

As a final note, the group also suggested that conferences on the issues 
in training graduate students would be worthwhile for a number of reasons^^^^ 
addition to comparability. 
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REFLECTIONS ON THE CONFERENCE 



Upon returning to their universities » many of the Conference participants 
wrote letters to the Panels expressing afterthoughts and suggestions concern*- 
Ing comparability In research. Excerpts from some of these letters are pre- 
sented below. In the hope that they might stimulate consideration of Issues 
beyond those raised In the Conference. The Panels would cojitlnue to welcome 
other reactions and Ideas both from those who participated In t|ie Conference 
and from those who read these proceedings. Letters could be addressed to the 
Panels through the Social Research Group, G.W.U. , 2401 Virginia Avenue, N.W. , 
20037. By providing Input In this way, researchers In the field could guide 
the Panels In their efforts to Increase comparability In research, and In 
could benefit from the general exch^^e of Ideas and Information that might 
result . 

• The Issue of cumulative research Is certainly an Important one, 
and I was happy to be part of a group considering It. Another 
series of groups composed of Individuals with common substantive 
research Interests should be able to go beyond agreeing In princi- 
ple .that the problem of comparability Is a real one and begin to 
make some more concrete suggestions as to vital demographic vari- 
ables and potential marker variables. 

— Kathleen M. White 
Boston University 

• I am at this time negotiating with the Graduate School to ascertain 
a date and the possibility of a conference here on campus. It 
would be delightful to have you and/or one of the project group to 
discuss the matter of comparability as It might eitlst on campus 

as well as on the national scene. 

— Rosemarle E. McCartln 
University of Washington 

9 Throughout both conferences I have attended, the comparability 

Issues have been discussed In relation to proposals. From looking 
at the guidelines for submitting a Fln4l R^ort to OCD, much com- 
parability could be achieved if the guidelines were to Include the 
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comparability issues. That way, all Final Reports of research 
sutxnitted to federal agencies would include coiiq)arability data. 
Were Final Reports to include the descripti^ of "marker variables 
and measures", however defined, it would increase the Government's 
ability to synthesize, utilize, and just know what it knows on ^ 
given research topic. 

— Jualynne Dodson 
Atlanta University 

Just a note to say that I found the recent conference on research 
comparability very interesting and instructive. While I developed 
a renewed respect for the necessity of achieving some generaliza- 
bility of findings, I became more appreciative as well of some of 
the possible scientific dangers involved in becoming obsessively 
concerned with comparability and the inclusion of marker variables. 
I think, for example, that the National Collaborative Perinatal 
Project tried to achieve comparability (with only limited success 
in some areas, incidentally) across 14 institutions in their respec- 
tive psychological, pediatric and other assessments, quite possibly 
to the detriment of other investments that should have been made 
in the study and in the study population at that time. 

— Lewis Lipsitt 
Bro^ University 

1. I got a rather schizophrenic feeling. For several years we have 
been hearing with increasing clarity messages from people like Bruner, 
Bronf enbrenner , Cole and myself that ecological factors are of pro- 
found importance in development and that moreover the ecology of the 
experimental situation is 6ften a major determinant of tested levels 
and styles of functioning. Accordingly, a proposal to establish 
comparability on the basis of demographic'f actors in the main runs 
counter to that view and seems to be somewhat regressive. 

2. Similarly, there has been increasing understanding that Standard 
tests are t|ot so sfandard as they are administered in different 
ecologies a^d to different cultural groups. Thus, the second aspect 
of comparabJJ.ity is open to question on the same grounds as the firsts 

3. Neither ot the above points argues against the attempt to establish 
comparability, but they do argue for the complexity of the issue. It 
seems clear chat marker vaj|jLables should th^selves be made objects 

of questioning rather thanTbeing assumed as known from the outset. The 
problem is not simple description, since each description functions 
like a theory \of what is important in accounting for the event. 

4. In order ti make some headway I think that two major efforts would 
be called for-4either of which I would be glad to be involved in. 
First, there should be conferences (not necessarily elite) about the 
psychological o^ descriptive meaningf ulness of those marker variables 
currently offere|^ for sale. Similarly there should be conferences 
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devoted to an attempt to develop a descriptive typoloRy of expert- ^ 
mental environments and data sources so as to give a balanced "marking" 
of the various research efforts considered for comparability. 

All of the above presumes that the Issue Is, In fact, a research 
Issue and that, accordingly, the best description of the comparability 
or non-comparablllty of experiments, etc. Is In fact best for the 
purpose ^t hand. However, none of the above makes sense If the 
effort Is a strategy of the research community to adapl^ Itself to the 
categories- of decision makers without a corresppndlng attempt to alter 
those categories. I bellreve that It would ultimately be Irreaponslble 
and counterproductive to adopt categories that we know to be false In 
order to save our funding skins. Hence, I hope that we are on the 
same side In wanting to see some Intellectually defensible set of ^ 
markers established that reflect current thinking In the field. To 
do so would Involve a comml^ent to study the Issue and to establish"^ ^ 
markers relevant to both subjects and situations. | 

— Joseph Click 

City University of New York 

I want to endorse the^^ldea that people from Washington go out to uni- 
versities to give talks or seminars on comparab'lllty , This Is an 
excellent way to get the message directly to graduate students and 
faculty. It also provides students and faculty a chance to meet Infor- 
mally with someone from the Washington scene and learn about the 
agency's perspective^ plans» and programs. . • . 

I urge you to consider the possibility of more extended conferences. 
Probably something like summer conferences of 3-6 weeks, located at 
some retreat-like setting, where work sessions arid day-to-day contact 
would be maintained. There are examples of this vehicle In psychology 
and developmental psychology. Several private foundations and govern- 
mental agencies have supported these conferences. 'I can see the con- 
tent and participants forming In several different directions. One 
could be a "boulder-llke" conference on training In developmental 
psychology. . . . 

Another possibility would be a conference on the topic of comparability 
of research. Participants could be researchers and directors of train- 
ing programs In btiman development. Presumably a volume reportlng^fce 
conference papers would emerge from the summer. 

A third possibility would be a summer Institute on comparability of 
research. In this format, advanced graduate students and recent doc- 
torates would be selected to attend the Institute and a faculty <would 
be assembled to teach 5 or 6 week courses on the topic. The faculty 
should Include government and university people. 

I believe that the USDA model of extension centers on various agri- 
cultural canxpukes to be a model worthy of serious study. One does 
not want to stop using current methods of supporting research and 
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training. However, I am convinced that supporting centers in the 
USDA manner at certain places could have enormous long-range benefits 
for social policy, applied and demonstration research, basic research 
and theory, and training. I should think that centers at widely dis- 
persed loeations, building upon local pools of talent, could work 
very well. Centers could be established in states such as: California, 
Minnesota, New York, Georgia, North Carolina, Texas, and Illinois. 
Using a university base in each state and grafting the center onto 
existing programs and personnel, yet maintaining the fundamental nature 
of the USDA model. 

— Martin Manosevitz 

University of Texas at Austin 
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FOLLOW-ON ACTIVITIES 



Participants: ' 

You may wish to know *what the Interagency Panels have been doing since 
the Conference, besides organzing the Proceedings, A number of things Jhave 
happened . 



^The^^anels have had the topic on each of their meeting agenda for June, 
July, and August. The discussions have been on what actions should be taken 



and what forms the actions should take to assure a continued thrust on the 
comparability, cumulative data/knowledge issues. From these discussions a 
consensus was reached that the issue of comparability and cumulative data/ 
kwwledge must continually be seen in the larger context of social policy and 



relevant research. Comparability is one sub-facet of the entire scope of 
research addressing social policy concerns. And, further, comparability 



problems can only be addressed depending on the state-of-the-art of a par- 
ticular concept, variable clarity and measurement. Many of you pointed some 
of these things out during the conference and in your subsequent letters to 
us. / 

As a result of the Panels' discussions, a paper was commissioned to 
Richard Bell. His paper became a discussion and work paper for Panel meetings 
and the model he presented was adopted, as modified, by both Panels. The 
adopted model is included in the Proceedings for your examination. It has 
become a guide for us and places research activities, including comparability 

and data/knowledge issues, in proper per spec l^ive. 

/ 

Presently, the Panels are determining which broad policy issues they 
wish to address on an interagency basis. Once these are selected, the Bell 
nodel will guide areas of action and discussion. One of these, you note, is 
a meeting of consumers and researchers to identify research questions appro- 
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prlate for addressing the socl^c policy. A State |kas already expressed a 
desire to use the entire^ State as a consumer of research and Is Interested In 
addressing research n^ds to carry out the social policy. Another area Is 

that of coinpar ability and cumulative data /knowledge. Here, again, meetings 

's 

will be held to discuss the state-of-the-art and needed actions. The Panels 
will use the data base of current research supported by the Federal Govern- 
ment for Identifying research relevant to selected social policy areas and . 
to analyze that research In terms^of Its status for further action. Including 
comparability and cumulative data /knowledge actions. 

We are also respoVidlng . to your requests to make presentatlmqa at your 
Institutions. And we look forward to a continuing relationship wlt>! you, 
either ^s an entire group pr as Individuals particularly expert and/or Inter- 
ested In research on the social policy areas selected by the Panels. We will 
keep you Informed. 




Edith H. Grotberg, Ph.D. 
Chairperson 

August 1975 
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This *ef fort will confine itself to comments or suggestions that appeared^^ 

in the transcripts of two or libre subgroups at the conferences. First of^ 
all, the conferees repeatedly accepted the necessity of addressing research 
to pressing polj^cy issues with which federal, state and other agenties, and 
the gineral public are concerned. There was no doubt but that the research 
establishment should (1) organize itself to produce answers to researchable 
questions entailed by these policy issues, and (2) assure that progress 
on researchable issues is achieved in a reasonable time relative to the 
urgency of the public need. 

/ 

Second, it was recommended that Consumer-Researcher Groups be organized 
to acquaint scientists with the nature of the problems involved in a policy 
question, and reciproj::ally , to communicate to consumer groups (representatives 
of agencies, or public groups that bear the brunt of the problems involved), 
the kind of assistance that may or may not be feasible in research. 

A third recommendation was that there be Research Progress Monitoring 
Groups that would, be responsible for keeping track of research relevant to 
certain key areas to determine whether reasonable progress has been made, 
or whether a number of approaches should be used to stimulate and facilitate 
the research. 

A fourth recommendation was that several kinds of conferences for 
researchers be organized, ranging from those intended to work on conceptual 
problems (so as to produce better^^ef initions in an area), through to 
conferences of potential collaborators having common research interests, to 
groups which have reached a stage where they could standardize the collection 
of demographic* information and develop marker variables. All of these 
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efforts were intended to provide for a more cumulative effect of individual 



accompanied by more complete descriptions of subjects, tests and procedures, 
and the setting, than would normally be published because of limitations of 
journal space • This supplementary! descriptive material (needed in order to 
tie results from different projects together) should be deposited in central 
data banks where it could be accessible to interested researchers. 

All of these recommendation^ could be integrated in a tripartite 
striiCture of interrelated groups designed to translate policy questions into 
research questions, and attain, answers to these questions as quickly as possible 

The first need in putting such a structure into effect, is for the 
Interagency Panels to collate and integrate policy questions submitted by 



agencies. The policy questions could come out of such problem areas as 
family breakdown, day care, violence in schools, runaway children, and 
child abuse. These policy questions could be addressed by Consumer- 
Researcher Groups which would be convened to explore various aspects and 
ramifications of the problem, to bring to bear the widest possible range of 
viewpoints, and open up as many research possibilities as possible. The 



Consumer-Researcher Groups might meet once or several times but would be 
considered to have discharged their function when they had outlined the aspects 
of the problem for which research of various kinds might be helpful, whether 
it be action-oriented research, evaluation research, or basic research 
designed to produce new information. Their recommendations would lead in 
some cases to the establishment ot Research Progress Monit0ring Groups. 
The#e groups would consist of experts in research for that area, who had 



research ^forts. 



A fifth recommendation was that submission to scientific journals be 
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a vital interest in obtaining answers to the questions raised by the 
Consumer-Researcher Groups, whether the questions lay in the areas of 

(1) action programs, (2) evaluation, or (3) obtaining new information. 
Thus, three different kinds of Research Progress Monitoring Groups could 
be seen. All of these groups would be continuing in nature and have long- 
term responsibilities relative to obtaining answers to the research questions 
that led to the creation of the group. They would be responsible for 
periodic progress reports. 

The operation of one of the three kinds of Research Progress Monitoring 
Groups could be projected by way of illustration. One intended to yield 
new research information could determine that progress in the area was 
sufficient without further stimulus to the field, and could discharge their 
function with periodic reports on results, and how they bore on the policy 
questions. On the other hand, it might determine that there was a need for 
better d^©graphic information, and request the cooperation of journal editors 
and others in obtaining information for a data bank that they would maintain. 
Alternatively, or in addition, it might be determined that the key need in 
this particular field at a given time was for conferences drawing together 
leading researchers, to assist them in clarifying concepts, and in develop- 
ing better theory. They could determine that research in the area had 
reaVhed a point at which it was very important that there be a variety of 
studies attempting to check out "the same or related findings; thus, they 
could encourage collaborative research involving a small number of investigators 
pursuing the same topics, and interested in working together. They might 
conclude that, because of insufficient use' of marker variables, it was not 
possible to synthesize the results of studies that had been carried out in 
the research area they were monitoring. Accordingly, they could recommend 
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to agencies or specific groups of agencies thatfAnvestigators be brought 
together to survey their area of research with the objective of developing 
marker variables as well as means' of measuring them. 

/ The purpose of the foregoing structure would be to provide a more certain 
linkage of research efforts to public policy questions, but not to replace 
research support ^or investigators who submit requests under the open ^ 
submission categories existing in many agencies and foundations. This 
schema carries no implications for a shift of refsearch support away from 
basic science. It is recognized that a great deal of progress in meeting 
public needs results from the efforts of individual scientists who themselves 
select what they consider to be the important problems and determine for 
themselves how to pursue them. Rather, the structure of these groups presently 
being recommended would ^e intended to prevent the kind of situation that 
Urie Bronfenbrenner has brought to the attention of investigators in 
developmental psychology in editorials and speeches at conventions. 
Bronfenbrenner has simply point^ed out that very little research has been carried 

^o^t^n some policy questions because of a lack of interest on the part of 
qualified scientists, or because the structure of their research system is 
such that there are no rewards to the investigators in pursuing answers to 
these public policy questions. In short, the purpose of the structure of 
research groups would be to insure that reasonable progress (of the kind 
that agencies, congressmen, and the public have a right to expect), will be 
forthcoming from the research establishment in return for the general support 
of research on human . development. 

It is clear that parts of the flow chart, from agency policy questions 
through to the convening of expert groups, are already in effect in some areas 
and for some problems. The idea of the Research Monitoring Groups is one 
of the more novel elements in the scheme. The main utility of the flow 
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chart l8 that it organizes a number of recommendations that have been made 
by very knowledgeable investigators that had been convened by the 
Interagency Panels • A large number of specific recommendations that 
can otherwise be treated only as a laundry list, find a place within this 
framework. The structure is also responsive to the repeated recommendation 
that solutions to research problems be pursued by area and that very general 
recommendations, whether they are concerned with better reporting of 
demographic variables or more use of co^aboratiye research, could be 
inappropriate for some research areas at best, or stultifying and counter- 
produ-ctive at the worst. Thus, this effort to integrate the high-frequency 
recommendations of the c6nferees places a responsibility for determing 
appropriate action in the hands of a group of research experts who would 
know better than anyone else what approaches are needed, and how a number 
of these apprgaches might have to be adapted for that particular area of 
research. / 
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